Don't Fear the Reaper: Refuting Bostrom's Superintelligence Argument

In recent years prominent intellectuals have raised ethical concerns about the consequences of artificial intelligence. One concern is that an autonomous agent might modify itself to become ”superintelligent” and, in supremely effective pursuit of poorly specified goals, destroy all of humanity. This paper considers and rejects the possibility of this outcome. We argue that this scenario depends on an agent’s ability to rapidly improve its ability to predict its environment through self-modification. Using a Bayesian model of a reasoning agent, we show that there are important limitations to how an agent may improve its predictive ability through self-modification alone. We conclude that concern about this artificial intelligence outcome is misplaced and better directed at policy questions around data access and storage. The appetite of the public and prominent intellectuals for the study of the ethical implications of artificial intelligence has increased in recent years. One captivating possibility is that artificial intelligence research might result in a ‘superintelligence’ that puts humanity at risk. (Russell 2014) has called for AI researchers to consider this possibility seriously because, however unlikely, its mere possibility is grave. (Bostrom 2014) argues for the importance of considering the risks of artificial intelligence as a research agenda. For Bostrom, the potential risks of artificial intelligence are not just at the scale of industrial mishaps or weapons of mass destruction. Rather, Bostrom argues that artificial intelligence has the potential to threaten humanity as a whole and determine the fate of the universe. We approach this grand thesis with a measure of skepticism. Nevertheless, we hope that by elucidating the argument and considering potential objections in good faith, we can get a better grip on the realistic ethical implications of artificial intelligence. This paper is in that spirit. We consider the argument for this AI doomsday scenario proposed by Bostrom (Bostrom 2014). Section 1 summarizes Bostrom’s argument and motivates the work of the rest of the paper. In focuses on the conditions of an “intelligence explosion” that would lead to a dominant machine intelligence averse to humanity. Section 2 argues that rather than speculating broadly about general artificial intelligence, we can predict outcomes of Copyright c © 2015, Sebastian Benthall. artificial intelligence by considering more narrowly a few tasks that are essential to instrumental reasoning. Section 3 considers recalcitrance, the resistance of a system to improvements to its own intelligence, and the ways it can limit intelligence explosion. Section 4 contains an analysis of the recalcitrance of prediction, using a Bayesian model of a predictive agent. We conclude that prediction is not something an agent can easily improve upon autonomously. Section 5 discusses the implication of these findings for further investigation into AI risk. 1 Bostrom’s core argument and definitions Bostrom makes a number of claims in the course of his argument which I will outline here as distinct propositions. Proposition 1. A system with sufficient intelligence relative to other intelligent systems will have a ‘decisive strategic advantage’ and will determine the fate of the world and uni-